Switch to PostgreSQL 16 image by ShimShtein · Pull Request #577 · theforeman/foremanctl

ShimShtein · 2026-06-18T13:24:54Z

Why are you introducing these changes? (Problem description, related links)

As part of being prepared to run on RHEL 10, we need to switch to postgres 16.

What are the changes introduced in this pull request?

Switch of the container to postgres 16 instead of 13.

How to test this pull request

If the system spins up successfully and works with content - it's good to go.

ShimShtein · 2026-06-18T13:28:40Z

Extracted out of #395

evgeni · 2026-06-18T14:04:53Z

Extracted out of #395

What for? (#395 already identified the problems with it)

ShimShtein · 2026-06-18T14:24:09Z

Because #395 has both postgres and valkey in the same PR?

ShimShtein · 2026-06-18T16:27:11Z

I see the tests are failing on password encryption algorithm detection: https://github.com/theforeman/foremanctl/actions/runs/27768839212/job/82162895170?pr=577#step:17:643
That's weird. Locally the same setup gives me:

[vagrant@quadlet ~]$ sudo podman exec postgresql psql -U postgres -c 'SHOW password_encryption'
 password_encryption 
---------------------
 scram-sha-256
(1 row)

Which is weird.

evgeni · 2026-06-22T09:41:41Z

I see the tests are failing on password encryption algorithm detection: https://github.com/theforeman/foremanctl/actions/runs/27768839212/job/82162895170?pr=577#step:17:643 That's weird. Locally the same setup gives me:
[vagrant@quadlet ~]$ sudo podman exec postgresql psql -U postgres -c 'SHOW password_encryption'
 password_encryption 
---------------------
 scram-sha-256
(1 row)
Which is weird.

It's not weird, as that was the ext DB and that wasn't using our image definition properly. Fixed.

ShimShtein · 2026-06-28T14:12:53Z

  ansible.builtin.include_role:
    name: debug_tools

+- name: 'Enable postgresql:16 module'


@evgeni, let me ask this question again: wouldn't it be more resilient to run the PG commands from inside a PG container instead of installing the client tooling directly on the host?

Would, sure. I just had none that I could integrate easily into the various workflows where we need Postgres tooling on the host.

You'd likely want to write a module that does this to be re-used because there are postgres actions that are run in a lot more places. I am not sure I see the value of it at the moment but it might be worth exploring after we get everything else.

@ehelms Debian 13 has PostgreSQL 17. That probably means you can't run pg_dump on the PostgreSQL 16 container we run, which breaks backups.

evgeni · 2026-06-29T12:43:10Z

The upgrade complains:

Jun 28 14:37:27 quadlet.example.com postgresql[123049]: Incompatible data directory.  This container image provides
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: PostgreSQL '16', but data directory is of
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: version '13'.
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: 
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: This image supports automatic data directory upgrade from
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: '15', please _carefully_ consult image documentation
Jun 28 14:37:27 quadlet.example.com postgresql[123049]: about how to use the '$POSTGRESQL_UPGRADE' startup option.

The fact its supporting 15→16 only is encoded here:
https://github.com/sclorg/postgresql-container/blob/f1c89bca212c6a829ab74fdfdeec264f0d08a992/16/Dockerfile.c10s#L13-L14

Funnily, the container actually contains 13 binaries:

% podman run --rm -ti quay.io/sclorg/postgresql-16-c10s bash
…
bash-5.2$ rpm -q postgresql-upgrade
postgresql-upgrade-16.13-2.el10.x86_64
bash-5.2$ rpm -qf /usr/lib64/pgsql/postgresql-13/bin/postgres
postgresql-upgrade-16.13-2.el10.x86_64
bash-5.2$ /usr/lib64/pgsql/postgresql-13/bin/postgres --version
postgres (PostgreSQL) 13.23

I'd argue this is a bug in the image we should ask for a fix.

ehelms · 2026-07-03T11:04:00Z

    - "../../../src/vars/defaults.yml"
    - "../../../src/vars/flavors/{{ flavor }}.yml"
    - "../../../src/vars/database.yml"
+    - "../../../src/vars/images.yml"


Why is this needed now?

ehelms · 2026-07-03T11:04:17Z

  notify:
    - Restart postgresql

+- name: Check if PG_VERSION file exists


This set of upgrade tasks look good. Could they go in a dedicated task file?

ehelms · 2026-07-03T11:04:43Z

+  ansible.builtin.systemd:
+    name: postgresql.service
+    masked: false
+  when:


As this when pattern is repeated, you can put it in a block.

ekohl

Wouldn't it be easier to unconditionally set POSTGRESQL_UPGRADE so it will always run when needed?

Thinking about the logging aspect: where do the logs of the upgrade go now? Can we somehow debug it when it fails? When it's part of the regular service then you'll have it the journal and sosreport will pick it up already.

ekohl · 2026-07-03T11:10:51Z

+      POSTGRESQL_MAX_CONNECTIONS: "{{ postgresql_max_connections }}"
+      POSTGRESQL_SHARED_BUFFERS: "{{ postgresql_shared_buffers }}"
+      POSTGRESQL_EFFECTIVE_CACHE_SIZE: "{{ postgresql_effective_cache_size }}"
+      POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"


Please link why this workaround is needed

Suggested change

POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"

# https://github.com/sclorg/postgresql-container/issues/661

POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"

ekohl · 2026-07-03T11:11:38Z

+      POSTGRESQL_SHARED_BUFFERS: "{{ postgresql_shared_buffers }}"
+      POSTGRESQL_EFFECTIVE_CACHE_SIZE: "{{ postgresql_effective_cache_size }}"
+      POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"
+      POSTGRESQL_UPGRADE: "hardlink"


Can you please make this a variable so we could easily switch to copy?

ekohl · 2026-07-03T11:17:58Z

+    - "postgresql_version == '13'"
+  register: postgresql_upgrade_result
+
+- name: Stop PostgreSQL systemd service (Quadlet)


Why is it needed to stop the service again?

ShimShtein · 2026-07-05T12:26:59Z

Wouldn't it be easier to unconditionally set POSTGRESQL_UPGRADE so it will always run when needed?

This will throw a warning: https://github.com/sclorg/postgresql-container/blob/134bb39ab5c435087487c3422522bddf160f727d/16/root/usr/share/container-scripts/postgresql/common.sh#L415 that will be visible in the logs (and if I am not mistaken it exits immediately, but at this point I am not sure if my environment was clean when I tested it). That was my original thought, but I had to work around the message.

Thinking about the logging aspect: where do the logs of the upgrade go now? Can we somehow debug it when it fails? When it's part of the regular service then you'll have it the journal and sosreport will pick it up already.

Had to debug the process quite a lot. You will have limited logging in the playbook itself (I will remove the debug log level from the playbook), but the more interesting log will be in the data folder {{ postgresql_data_dir }}/userdata/log/ and {{ postgresql_data_dir }}/userdata-new/log/ depending on the failure stage. We do not collect these files explicitly in the sosreport, it is a bit tricky - I would say that it should be part of the postgresql plugin, since it's a standard postgresql action. On the other hand, the files are created there due to our specific postgresql configuration, so we can argue that we should collect this data. Let's create an issue and track postgresql upgrade logs separately.

ShimShtein · 2026-07-06T15:18:02Z

packit error seems unrelated:

2026-07-06 12:17:47.459 logging.py        INFO   HTTP ERROR response 429  [https://raw.githubusercontent.com/theforeman/foreman-packaging/rpm/develop/packages/foreman/foremanctl/foremanctl.spec]

ekohl

This comment is not blocking, but something to consider. Possibly in a follow up.

ekohl · 2026-07-06T10:48:32Z

+      containers.podman.podman_container:
+        name: "{{ postgresql_container_name }}-upgrade"
+        image: postgresql.image
+        state: quadlet


Does it need to be a quadlet? In favor of it means you get logs in the journal, but it's also fragile: if Ansible somehow starts, it can leave the system in an unpredictable state.

if Ansible somehow starts, it can leave the system in an unpredictable state.

I don't follow this logic. However, I assume you are asking the question of a quadlet which generates a systemd service vs. just running the container directly with the equivalent of a podman run?

Does it need to be a quadlet?

I had the same question. I decided to go on with a quadlet to reuse the postgres.image container. If I spin it up as a separate container, it means I cannot reuse the image definition and any drop-ins that will be defined there. Originally it was a regular container.

And I messed up: I intended to write "Ansible somehow stops in the middle". Not start.

I don't think it is disastrous:

if the playbook stops before defining the service - no harm done

if it stops after the service is defined, but before it executes the service - it will be executed at first reboot

if it stops after the execution - it may leave some leftovers, but they would be harmless

There are options to add the upgrade as a dependency for the postgres.container, so it will pull the service before the execution and then push the condition into the service startup, so it will just silently finish with noop in case the DB was already migrated.

I think it's too complex for a one time only step.

ekohl

I would be OK with tracking the items in a follow up task when someone actually does proper testing of upgrades. Right now I consider this "good enough" to make CI pass.

ekohl · 2026-07-07T12:46:02Z

+        quadlet_options:
+          - |
+            [Unit]
+            Description=PostgreSQL Database Upgrade from 13 to 16


Perhaps make sure that systemd knows it's 1 or the other? Or will this introduce too much uncertainty?

Suggested change

Description=PostgreSQL Database Upgrade from 13 to 16

Description=PostgreSQL Database Upgrade from 13 to 16

Conflicts=postgresql.service

According to Gemini, the directive is symmetric, so it won't protect us from someone that tries to start the server. Actually it will just cause the upgrade container to stop (even more problematic in the middle of an upgrade).

ekohl · 2026-07-07T12:48:10Z

+            Description=PostgreSQL Database Upgrade from 13 to 16
+            [Service]
+            Type=oneshot
+            RemainAfterExit=yes


This means the service will remain as active. Is that useful?

It will mark the service as active(exited) when it will stop the execution. In the happy path it won't matter, since we remove the service completely, but if it fails, we will have indication whether it was run or not.

pr-processor Bot added the Not yet reviewed label Jun 18, 2026

ShimShtein marked this pull request as ready for review June 18, 2026 13:25

ShimShtein force-pushed the test_pg16 branch from 678ef27 to a24bcfa Compare June 18, 2026 15:03

evgeni force-pushed the test_pg16 branch 2 times, most recently from f9dc89d to f6f88a0 Compare June 22, 2026 09:40

evgeni force-pushed the test_pg16 branch from f6f88a0 to 14e033e Compare June 22, 2026 12:13

evgeni mentioned this pull request Jun 23, 2026

Use different foremanctl versions when testing upgrades #586

Merged

2 tasks

ShimShtein force-pushed the test_pg16 branch from 14e033e to 2c2c937 Compare June 28, 2026 14:05

Switch to postgres 16 image

08e4391

ShimShtein force-pushed the test_pg16 branch from 2c2c937 to 08e4391 Compare June 28, 2026 14:11

ShimShtein commented Jun 28, 2026

View reviewed changes

pr-processor Bot removed the Not yet reviewed label Jun 28, 2026

evgeni closed this Jun 29, 2026

evgeni reopened this Jun 29, 2026

ShimShtein closed this Jul 1, 2026

ShimShtein reopened this Jul 1, 2026

ekohl mentioned this pull request Jul 1, 2026

PostgreSQL 16 images based on EL10 have an incorrect POSTGRESQL_PREV_VERSION set sclorg/postgresql-container#661

Open

ShimShtein force-pushed the test_pg16 branch 2 times, most recently from 72e983b to 7e0aadc Compare July 2, 2026 23:02

ehelms reviewed Jul 3, 2026

View reviewed changes

ekohl changed the title ~~Switch to postgres 16 image~~ Switch to PostgreSQL 16 image Jul 3, 2026

ekohl reviewed Jul 3, 2026

View reviewed changes

Add a data migration steps

83a028d

ShimShtein force-pushed the test_pg16 branch from 7e0aadc to 0a52fde Compare July 5, 2026 17:52

evgeni reviewed Jul 6, 2026

View reviewed changes

Comment thread src/roles/postgresql/tasks/upgrade.yaml

Switch to quadlet to reuse the postgresql.image definition

76f2858

ShimShtein force-pushed the test_pg16 branch from 0a52fde to 76f2858 Compare July 6, 2026 09:57

Stop all the foreman.target instead of just PG

3cc182f

evgeni approved these changes Jul 6, 2026

View reviewed changes

ekohl reviewed Jul 6, 2026

View reviewed changes

ekohl reviewed Jul 7, 2026

View reviewed changes

	POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"
	# https://github.com/sclorg/postgresql-container/issues/661
	POSTGRESQL_PREV_VERSION: "{{ postgresql_version }}"

	Description=PostgreSQL Database Upgrade from 13 to 16
	Description=PostgreSQL Database Upgrade from 13 to 16
	Conflicts=postgresql.service

Uh oh!

Conversation

ShimShtein commented Jun 18, 2026

Why are you introducing these changes? (Problem description, related links)

What are the changes introduced in this pull request?

How to test this pull request

Uh oh!

ShimShtein commented Jun 18, 2026

Uh oh!

evgeni commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShimShtein commented Jun 18, 2026

Uh oh!

ShimShtein commented Jun 18, 2026

Uh oh!

evgeni commented Jun 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evgeni commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ekohl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ShimShtein commented Jul 5, 2026

Uh oh!

Uh oh!

ShimShtein commented Jul 6, 2026

Uh oh!

ekohl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ekohl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

evgeni commented Jun 18, 2026 •

edited

Loading

evgeni commented Jun 29, 2026 •

edited

Loading